

<!DOCTYPE html>
<html class="writer-html5" lang="en" >
<head>
  <meta charset="utf-8" />
  
  <meta name="viewport" content="width=device-width, initial-scale=1.0" />
  
  <title>Cloud Transition &mdash; Ceph Documentation</title>
  

  
  <link rel="stylesheet" href="../../_static/ceph.css" type="text/css" />
  <link rel="stylesheet" href="../../_static/pygments.css" type="text/css" />
  <link rel="stylesheet" href="../../_static/graphviz.css" type="text/css" />
  <link rel="stylesheet" href="../../_static/css/custom.css" type="text/css" />

  
  
    <link rel="shortcut icon" href="../../_static/favicon.ico"/>
  

  
  

  

  
  <!--[if lt IE 9]>
    <script src="../../_static/js/html5shiv.min.js"></script>
  <![endif]-->
  
    
      <script type="text/javascript" id="documentation_options" data-url_root="../../" src="../../_static/documentation_options.js"></script>
        <script src="../../_static/jquery.js"></script>
        <script src="../../_static/underscore.js"></script>
        <script src="../../_static/doctools.js"></script>
    
    <script type="text/javascript" src="../../_static/js/theme.js"></script>

    
    <link rel="index" title="Index" href="../../genindex/" />
    <link rel="search" title="Search" href="../../search/" />
    <link rel="next" title="Ceph 管理器守护进程" href="../../mgr/" />
    <link rel="prev" title="D3N RGW Data Cache" href="../d3n_datacache/" /> 
</head>

<body class="wy-body-for-nav">

   
  <header class="top-bar">
    

















<div role="navigation" aria-label="breadcrumbs navigation">

  <ul class="wy-breadcrumbs">
    
      <li><a href="../../" class="icon icon-home"></a> &raquo;</li>
        
          <li><a href="../">Ceph 对象网关</a> &raquo;</li>
        
      <li>Cloud Transition</li>
    
    
      <li class="wy-breadcrumbs-aside">
        
          
            <a href="../../_sources/radosgw/cloud-transition.rst.txt" rel="nofollow"> View page source</a>
          
        
      </li>
    
  </ul>

  
  <hr/>
</div>
  </header>
  <div class="wy-grid-for-nav">
    
    <nav data-toggle="wy-nav-shift" class="wy-nav-side">
      <div class="wy-side-scroll">
        <div class="wy-side-nav-search"  style="background: #eee" >
          

          
            <a href="../../">
          

          
            
            <img src="../../_static/logo.png" class="logo" alt="Logo"/>
          
          </a>

          

          
<div role="search">
  <form id="rtd-search-form" class="wy-form" action="../../search/" method="get">
    <input type="text" name="q" placeholder="Search docs" />
    <input type="hidden" name="check_keywords" value="yes" />
    <input type="hidden" name="area" value="default" />
  </form>
</div>

          
        </div>

        
        <div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="main navigation">
          
            
            
              
            
            
              <ul class="current">
<li class="toctree-l1"><a class="reference internal" href="../../start/intro/">Ceph 简介</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../install/">安装 Ceph</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../cephadm/">Cephadm</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../rados/">Ceph 存储集群</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../cephfs/">Ceph 文件系统</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../rbd/">Ceph 块设备</a></li>
<li class="toctree-l1 current"><a class="reference internal" href="../">Ceph 对象网关</a><ul class="current">
<li class="toctree-l2"><a class="reference internal" href="../frontends/">HTTP 前端</a></li>
<li class="toctree-l2"><a class="reference internal" href="../placement/">存储池归置与存储类</a></li>
<li class="toctree-l2"><a class="reference internal" href="../multisite/">多站配置</a></li>
<li class="toctree-l2"><a class="reference internal" href="../multisite-sync-policy/">多站同步策略配置</a></li>
<li class="toctree-l2"><a class="reference internal" href="../pools/">存储池的配置</a></li>
<li class="toctree-l2"><a class="reference internal" href="../config-ref/">配置参考</a></li>
<li class="toctree-l2"><a class="reference internal" href="../admin/">管理指南</a></li>
<li class="toctree-l2"><a class="reference internal" href="../s3/">S3 API</a></li>
<li class="toctree-l2"><a class="reference internal" href="../rgw-cache/">Data caching and CDN</a></li>
<li class="toctree-l2"><a class="reference internal" href="../swift/">Swift API</a></li>
<li class="toctree-l2"><a class="reference internal" href="../adminops/">管理操作 API</a></li>
<li class="toctree-l2"><a class="reference internal" href="../api/">Python 接口</a></li>
<li class="toctree-l2"><a class="reference internal" href="../nfs/">通过 NFS 导出</a></li>
<li class="toctree-l2"><a class="reference internal" href="../keystone/">与 OpenStack Keystone 对接</a></li>
<li class="toctree-l2"><a class="reference internal" href="../barbican/">与 OpenStack Barbican 对接</a></li>
<li class="toctree-l2"><a class="reference internal" href="../vault/">与 HashiCorp Vault 对接</a></li>
<li class="toctree-l2"><a class="reference internal" href="../kmip/">KMIP Integration</a></li>
<li class="toctree-l2"><a class="reference internal" href="../opa/">与 Open Policy Agent 对接</a></li>
<li class="toctree-l2"><a class="reference internal" href="../multitenancy/">多租户</a></li>
<li class="toctree-l2"><a class="reference internal" href="../compression/">压缩</a></li>
<li class="toctree-l2"><a class="reference internal" href="../ldap-auth/">LDAP 认证</a></li>
<li class="toctree-l2"><a class="reference internal" href="../encryption/">服务器端加密</a></li>
<li class="toctree-l2"><a class="reference internal" href="../bucketpolicy/">桶策略</a></li>
<li class="toctree-l2"><a class="reference internal" href="../dynamicresharding/">动态的桶索引重分片</a></li>
<li class="toctree-l2"><a class="reference internal" href="../mfa/">多因子认证</a></li>
<li class="toctree-l2"><a class="reference internal" href="../sync-modules/">同步模块</a></li>
<li class="toctree-l2"><a class="reference internal" href="../notifications/">Bucket Notifications</a></li>
<li class="toctree-l2"><a class="reference internal" href="../layout/">RADOS 中的数据布局</a></li>
<li class="toctree-l2"><a class="reference internal" href="../STS/">STS</a></li>
<li class="toctree-l2"><a class="reference internal" href="../STSLite/">STS Lite</a></li>
<li class="toctree-l2"><a class="reference internal" href="../keycloak/">Keycloak</a></li>
<li class="toctree-l2"><a class="reference internal" href="../role/">Role</a></li>
<li class="toctree-l2"><a class="reference internal" href="../session-tags/">Session Tags</a></li>
<li class="toctree-l2"><a class="reference internal" href="../orphans/">Orphan List and Associated Tooliing</a></li>
<li class="toctree-l2"><a class="reference internal" href="../oidc/">OpenID Connect Provider</a></li>
<li class="toctree-l2"><a class="reference internal" href="../troubleshooting/">故障排除</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../man/8/radosgw/">radosgw 手册页</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../man/8/radosgw-admin/">radosgw-admin 手册页</a></li>
<li class="toctree-l2"><a class="reference internal" href="../qat-accel/">使用 QAT 为加密和压缩提速</a></li>
<li class="toctree-l2"><a class="reference internal" href="../s3select/">S3-select</a></li>
<li class="toctree-l2"><a class="reference internal" href="../lua-scripting/">Lua Scripting</a></li>
<li class="toctree-l2"><a class="reference internal" href="../d3n_datacache/">D3N Data Cache</a></li>
<li class="toctree-l2 current"><a class="current reference internal" href="#">Cloud Transition</a><ul>
<li class="toctree-l3"><a class="reference internal" href="#cloud-storage-class-configuration">Cloud Storage Class Configuration</a><ul>
<li class="toctree-l4"><a class="reference internal" href="#cloud-transition-specific-configurables">Cloud Transition Specific Configurables:</a></li>
<li class="toctree-l4"><a class="reference internal" href="#s3-specific-configurables">S3 Specific Configurables:</a></li>
<li class="toctree-l4"><a class="reference internal" href="#how-to-configure">How to Configure</a></li>
</ul>
</li>
<li class="toctree-l3"><a class="reference internal" href="#object-modification-limitations">Object modification &amp; Limitations</a><ul>
<li class="toctree-l4"><a class="reference internal" href="#versioned-objects">Versioned Objects</a></li>
</ul>
</li>
<li class="toctree-l3"><a class="reference internal" href="#future-work">Future Work</a></li>
</ul>
</li>
</ul>
</li>
<li class="toctree-l1"><a class="reference internal" href="../../mgr/">Ceph 管理器守护进程</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../mgr/dashboard/">Ceph 仪表盘</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../api/">API 文档</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../architecture/">体系结构</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../dev/developer_guide/">开发者指南</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../dev/internals/">Ceph 内幕</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../governance/">项目管理</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../foundation/">Ceph 基金会</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../ceph-volume/">ceph-volume</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../releases/general/">Ceph 版本（总目录）</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../releases/">Ceph 版本（索引）</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../security/">Security</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../glossary/">Ceph 术语</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../jaegertracing/">Tracing</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../translation_cn/">中文版翻译资源</a></li>
</ul>

            
          
        </div>
        
      </div>
    </nav>

    <section data-toggle="wy-nav-shift" class="wy-nav-content-wrap">

      
      <nav class="wy-nav-top" aria-label="top navigation">
        
          <i data-toggle="wy-nav-top" class="fa fa-bars"></i>
          <a href="../../">Ceph</a>
        
      </nav>


      <div class="wy-nav-content">
        
        <div class="rst-content">
        
          <div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
           <div itemprop="articleBody">
            
<div id="dev-warning" class="admonition note">
  <p class="first admonition-title">Notice</p>
  <p class="last">This document is for a development version of Ceph.</p>
</div>
  <div id="docubetter" align="right" style="padding: 5px; font-weight: bold;">
    <a href="https://pad.ceph.com/p/Report_Documentation_Bugs">Report a Documentation Bug</a>
  </div>

  
  <div class="section" id="cloud-transition">
<h1>Cloud Transition<a class="headerlink" href="#cloud-transition" title="Permalink to this headline">¶</a></h1>
<p>This feature enables data transition to a remote cloud service as part of <a class="reference external" href="https://docs.aws.amazon.com/AmazonS3/latest/dev/object-lifecycle-mgmt.html">Lifecycle Configuration</a> via <a class="reference internal" href="../placement/#storage-classes"><span class="std std-ref">存储类</span></a>. The transition is unidirectional; data cannot be transitioned back from the remote zone. The goal of this feature is to enable data transition to multiple cloud providers. The currently supported cloud providers are those that are compatible with AWS (S3).</p>
<p>Special storage class of tier type <code class="docutils literal notranslate"><span class="pre">cloud-s3</span></code> is used to configure the remote cloud S3 object store service to which the data needs to be transitioned. These are defined in terms of zonegroup placement targets and unlike regular storage classes, do not need a data pool.</p>
<p>User credentials for the remote cloud object store service need to be configured. Note that source ACLs will not
be preserved. It is possible to map permissions of specific source users to specific destination users.</p>
<div class="section" id="cloud-storage-class-configuration">
<h2>Cloud Storage Class Configuration<a class="headerlink" href="#cloud-storage-class-configuration" title="Permalink to this headline">¶</a></h2>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="p">{</span>
  <span class="s2">&quot;access_key&quot;</span><span class="p">:</span> <span class="o">&lt;</span><span class="n">access</span><span class="o">&gt;</span><span class="p">,</span>
  <span class="s2">&quot;secret&quot;</span><span class="p">:</span> <span class="o">&lt;</span><span class="n">secret</span><span class="o">&gt;</span><span class="p">,</span>
  <span class="s2">&quot;endpoint&quot;</span><span class="p">:</span> <span class="o">&lt;</span><span class="n">endpoint</span><span class="o">&gt;</span><span class="p">,</span>
  <span class="s2">&quot;region&quot;</span><span class="p">:</span> <span class="o">&lt;</span><span class="n">region</span><span class="o">&gt;</span><span class="p">,</span>
  <span class="s2">&quot;host_style&quot;</span><span class="p">:</span> <span class="o">&lt;</span><span class="n">path</span> <span class="o">|</span> <span class="n">virtual</span><span class="o">&gt;</span><span class="p">,</span>
  <span class="s2">&quot;acls&quot;</span><span class="p">:</span> <span class="p">[</span> <span class="p">{</span> <span class="s2">&quot;type&quot;</span><span class="p">:</span> <span class="o">&lt;</span><span class="nb">id</span> <span class="o">|</span> <span class="n">email</span> <span class="o">|</span> <span class="n">uri</span><span class="o">&gt;</span><span class="p">,</span>
              <span class="s2">&quot;source_id&quot;</span><span class="p">:</span> <span class="o">&lt;</span><span class="n">source_id</span><span class="o">&gt;</span><span class="p">,</span>
              <span class="s2">&quot;dest_id&quot;</span><span class="p">:</span> <span class="o">&lt;</span><span class="n">dest_id</span><span class="o">&gt;</span> <span class="p">}</span> <span class="o">...</span> <span class="p">],</span>
  <span class="s2">&quot;target_path&quot;</span><span class="p">:</span> <span class="o">&lt;</span><span class="n">target_path</span><span class="o">&gt;</span><span class="p">,</span>
  <span class="s2">&quot;target_storage_class&quot;</span><span class="p">:</span> <span class="o">&lt;</span><span class="n">target</span><span class="o">-</span><span class="n">storage</span><span class="o">-</span><span class="n">class</span><span class="o">&gt;</span><span class="p">,</span>
  <span class="s2">&quot;multipart_sync_threshold&quot;</span><span class="p">:</span> <span class="p">{</span><span class="n">object_size</span><span class="p">},</span>
  <span class="s2">&quot;multipart_min_part_size&quot;</span><span class="p">:</span> <span class="p">{</span><span class="n">part_size</span><span class="p">},</span>
  <span class="s2">&quot;retain_head_object&quot;</span><span class="p">:</span> <span class="o">&lt;</span><span class="n">true</span> <span class="o">|</span> <span class="n">false</span><span class="o">&gt;</span>
<span class="p">}</span>
</pre></div>
</div>
<div class="section" id="cloud-transition-specific-configurables">
<h3>Cloud Transition Specific Configurables:<a class="headerlink" href="#cloud-transition-specific-configurables" title="Permalink to this headline">¶</a></h3>
<ul class="simple">
<li><p><code class="docutils literal notranslate"><span class="pre">access_key</span></code> (string)</p></li>
</ul>
<p>The remote cloud S3 access key that will be used for a specific connection.</p>
<ul class="simple">
<li><p><code class="docutils literal notranslate"><span class="pre">secret</span></code> (string)</p></li>
</ul>
<p>The secret key for the remote cloud S3 service.</p>
<ul class="simple">
<li><p><code class="docutils literal notranslate"><span class="pre">endpoint</span></code> (string)</p></li>
</ul>
<p>URL of remote cloud S3 service endpoint.</p>
<ul class="simple">
<li><p><code class="docutils literal notranslate"><span class="pre">region</span></code> (string)</p></li>
</ul>
<p>The remote cloud S3 service region name.</p>
<ul class="simple">
<li><p><code class="docutils literal notranslate"><span class="pre">host_style</span></code> (path | virtual)</p></li>
</ul>
<p>Type of host style to be used when accessing remote cloud S3 endpoint (default: <code class="docutils literal notranslate"><span class="pre">path</span></code>).</p>
<ul class="simple">
<li><p><code class="docutils literal notranslate"><span class="pre">acls</span></code> (array)</p></li>
</ul>
<p>Contains a list of <code class="docutils literal notranslate"><span class="pre">acl_mappings</span></code>.</p>
<ul class="simple">
<li><p><code class="docutils literal notranslate"><span class="pre">acl_mapping</span></code> (container)</p></li>
</ul>
<p>Each <code class="docutils literal notranslate"><span class="pre">acl_mapping</span></code> structure contains <code class="docutils literal notranslate"><span class="pre">type</span></code>, <code class="docutils literal notranslate"><span class="pre">source_id</span></code>, and <code class="docutils literal notranslate"><span class="pre">dest_id</span></code>. These
will define the ACL mutation that will be done on each object. An ACL mutation allows converting source
user id to a destination id.</p>
<ul class="simple">
<li><p><code class="docutils literal notranslate"><span class="pre">type</span></code> (id | email | uri)</p></li>
</ul>
<p>ACL type: <code class="docutils literal notranslate"><span class="pre">id</span></code> defines user id, <code class="docutils literal notranslate"><span class="pre">email</span></code> defines user by email, and <code class="docutils literal notranslate"><span class="pre">uri</span></code> defines user by <code class="docutils literal notranslate"><span class="pre">uri</span></code> (group).</p>
<ul class="simple">
<li><p><code class="docutils literal notranslate"><span class="pre">source_id</span></code> (string)</p></li>
</ul>
<p>ID of user in the source zone.</p>
<ul class="simple">
<li><p><code class="docutils literal notranslate"><span class="pre">dest_id</span></code> (string)</p></li>
</ul>
<p>ID of user in the destination.</p>
<ul class="simple">
<li><p><code class="docutils literal notranslate"><span class="pre">target_path</span></code> (string)</p></li>
</ul>
<p>A string that defines how the target path is created. The target path specifies a prefix to which
the source ‘bucket-name/object-name’ is appended. If not specified the target_path created is “rgwx-${zonegroup}-${storage-class}-cloud-bucket”.</p>
<p>For example: <code class="docutils literal notranslate"><span class="pre">target_path</span> <span class="pre">=</span> <span class="pre">rgwx-archive-${zonegroup}/</span></code></p>
<ul class="simple">
<li><p><code class="docutils literal notranslate"><span class="pre">target_storage_class</span></code> (string)</p></li>
</ul>
<p>A string that defines the target storage class to which the object transitions to. If not specified, object is transitioned to STANDARD storage class.</p>
<ul class="simple">
<li><p><code class="docutils literal notranslate"><span class="pre">retain_head_object</span></code> (true | false)</p></li>
</ul>
<p>If true, retains the metadata of the object transitioned to cloud. If false (default), the object is deleted post transition.
This option is ignored for current versioned objects. For more details, refer to section “Versioned Objects” below.</p>
</div>
<div class="section" id="s3-specific-configurables">
<h3>S3 Specific Configurables:<a class="headerlink" href="#s3-specific-configurables" title="Permalink to this headline">¶</a></h3>
<p>Currently cloud transition will only work with backends that are compatible with AWS S3. There are
a few configurables that can be used to tweak its behavior when accessing these cloud services:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="p">{</span>
  <span class="s2">&quot;multipart_sync_threshold&quot;</span><span class="p">:</span> <span class="p">{</span><span class="n">object_size</span><span class="p">},</span>
  <span class="s2">&quot;multipart_min_part_size&quot;</span><span class="p">:</span> <span class="p">{</span><span class="n">part_size</span><span class="p">}</span>
<span class="p">}</span>
</pre></div>
</div>
<ul class="simple">
<li><p><code class="docutils literal notranslate"><span class="pre">multipart_sync_threshold</span></code> (integer)</p></li>
</ul>
<p>Objects this size or larger will be transitioned to the cloud using multipart upload.</p>
<ul class="simple">
<li><p><code class="docutils literal notranslate"><span class="pre">multipart_min_part_size</span></code> (integer)</p></li>
</ul>
<p>Minimum parts size to use when transitioning objects using multipart upload.</p>
</div>
<div class="section" id="how-to-configure">
<h3>How to Configure<a class="headerlink" href="#how-to-configure" title="Permalink to this headline">¶</a></h3>
<p>See <a class="reference internal" href="../placement/#adding-a-storage-class"><span class="std std-ref">新增一个存储类</span></a> for how to configure storage-class for a zonegroup. The cloud transition requires a creation of a special storage class with tier type defined as <code class="docutils literal notranslate"><span class="pre">cloud-s3</span></code></p>
<div class="admonition note">
<p class="admonition-title">Note</p>
<p>If you have not done any previous <a class="reference external" href="../multisite">Multisite Configuration</a>,
a <code class="docutils literal notranslate"><span class="pre">default</span></code> zone and zonegroup are created for you, and changes
to the zone/zonegroup will not take effect until the Ceph Object
Gateways are restarted. If you have created a realm for multisite,
the zone/zonegroup changes will take effect once the changes are
committed with <code class="docutils literal notranslate"><span class="pre">radosgw-admin</span> <span class="pre">period</span> <span class="pre">update</span> <span class="pre">--commit</span></code>.</p>
</div>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="c1"># radosgw-admin zonegroup placement add --rgw-zonegroup={zone-group-name} \</span>
                                        <span class="o">--</span><span class="n">placement</span><span class="o">-</span><span class="nb">id</span><span class="o">=</span><span class="p">{</span><span class="n">placement</span><span class="o">-</span><span class="nb">id</span><span class="p">}</span> \
                                        <span class="o">--</span><span class="n">storage</span><span class="o">-</span><span class="n">class</span><span class="o">=</span><span class="p">{</span><span class="n">storage</span><span class="o">-</span><span class="n">class</span><span class="o">-</span><span class="n">name</span><span class="p">}</span> \
                                        <span class="o">--</span><span class="n">tier</span><span class="o">-</span><span class="nb">type</span><span class="o">=</span><span class="n">cloud</span><span class="o">-</span><span class="n">s3</span>
</pre></div>
</div>
<p>For example:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="c1"># radosgw-admin zonegroup placement add --rgw-zonegroup=default \</span>
                                        <span class="o">--</span><span class="n">placement</span><span class="o">-</span><span class="nb">id</span><span class="o">=</span><span class="n">default</span><span class="o">-</span><span class="n">placement</span> \
                                        <span class="o">--</span><span class="n">storage</span><span class="o">-</span><span class="n">class</span><span class="o">=</span><span class="n">CLOUDTIER</span> <span class="o">--</span><span class="n">tier</span><span class="o">-</span><span class="nb">type</span><span class="o">=</span><span class="n">cloud</span><span class="o">-</span><span class="n">s3</span>
<span class="p">[</span>
    <span class="p">{</span>
        <span class="s2">&quot;key&quot;</span><span class="p">:</span> <span class="s2">&quot;default-placement&quot;</span><span class="p">,</span>
        <span class="s2">&quot;val&quot;</span><span class="p">:</span> <span class="p">{</span>
            <span class="s2">&quot;name&quot;</span><span class="p">:</span> <span class="s2">&quot;default-placement&quot;</span><span class="p">,</span>
            <span class="s2">&quot;tags&quot;</span><span class="p">:</span> <span class="p">[],</span>
            <span class="s2">&quot;storage_classes&quot;</span><span class="p">:</span> <span class="p">[</span>
                <span class="s2">&quot;CLOUDTIER&quot;</span><span class="p">,</span>
                <span class="s2">&quot;STANDARD&quot;</span>
            <span class="p">],</span>
            <span class="s2">&quot;tier_targets&quot;</span><span class="p">:</span> <span class="p">[</span>
                <span class="p">{</span>
                    <span class="s2">&quot;key&quot;</span><span class="p">:</span> <span class="s2">&quot;CLOUDTIER&quot;</span><span class="p">,</span>
                    <span class="s2">&quot;val&quot;</span><span class="p">:</span> <span class="p">{</span>
                        <span class="s2">&quot;tier_type&quot;</span><span class="p">:</span> <span class="s2">&quot;cloud-s3&quot;</span><span class="p">,</span>
                        <span class="s2">&quot;storage_class&quot;</span><span class="p">:</span> <span class="s2">&quot;CLOUDTIER&quot;</span><span class="p">,</span>
                        <span class="s2">&quot;retain_head_object&quot;</span><span class="p">:</span> <span class="s2">&quot;false&quot;</span><span class="p">,</span>
                        <span class="s2">&quot;s3&quot;</span><span class="p">:</span> <span class="p">{</span>
                            <span class="s2">&quot;endpoint&quot;</span><span class="p">:</span> <span class="s2">&quot;&quot;</span><span class="p">,</span>
                            <span class="s2">&quot;access_key&quot;</span><span class="p">:</span> <span class="s2">&quot;&quot;</span><span class="p">,</span>
                            <span class="s2">&quot;secret&quot;</span><span class="p">:</span> <span class="s2">&quot;&quot;</span><span class="p">,</span>
                            <span class="s2">&quot;host_style&quot;</span><span class="p">:</span> <span class="s2">&quot;path&quot;</span><span class="p">,</span>
                            <span class="s2">&quot;target_storage_class&quot;</span><span class="p">:</span> <span class="s2">&quot;&quot;</span><span class="p">,</span>
                            <span class="s2">&quot;target_path&quot;</span><span class="p">:</span> <span class="s2">&quot;&quot;</span><span class="p">,</span>
                            <span class="s2">&quot;acl_mappings&quot;</span><span class="p">:</span> <span class="p">[],</span>
                            <span class="s2">&quot;multipart_sync_threshold&quot;</span><span class="p">:</span> <span class="mi">33554432</span><span class="p">,</span>
                            <span class="s2">&quot;multipart_min_part_size&quot;</span><span class="p">:</span> <span class="mi">33554432</span>
                        <span class="p">}</span>
                    <span class="p">}</span>
                <span class="p">}</span>
            <span class="p">]</span>
        <span class="p">}</span>
    <span class="p">}</span>
<span class="p">]</span>
</pre></div>
</div>
<div class="admonition note">
<p class="admonition-title">Note</p>
<p>Once a storage class is created of <code class="docutils literal notranslate"><span class="pre">--tier-type=cloud-s3</span></code>, it cannot be later modified to any other storage class type.</p>
</div>
<p>The tier configuration can be then done using the following command</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="c1"># radosgw-admin zonegroup placement modify --rgw-zonegroup={zone-group-name} \</span>
                                           <span class="o">--</span><span class="n">placement</span><span class="o">-</span><span class="nb">id</span><span class="o">=</span><span class="p">{</span><span class="n">placement</span><span class="o">-</span><span class="nb">id</span><span class="p">}</span> \
                                           <span class="o">--</span><span class="n">storage</span><span class="o">-</span><span class="n">class</span><span class="o">=</span><span class="p">{</span><span class="n">storage</span><span class="o">-</span><span class="n">class</span><span class="o">-</span><span class="n">name</span><span class="p">}</span> \
                                           <span class="o">--</span><span class="n">tier</span><span class="o">-</span><span class="n">config</span><span class="o">=</span><span class="p">{</span><span class="n">key</span><span class="p">}</span><span class="o">=</span><span class="p">{</span><span class="n">val</span><span class="p">}[,{</span><span class="n">key</span><span class="p">}</span><span class="o">=</span><span class="p">{</span><span class="n">val</span><span class="p">}]</span>
</pre></div>
</div>
<p>The <code class="docutils literal notranslate"><span class="pre">key</span></code> in the configuration specifies the config variable that needs to be updated, and
the <code class="docutils literal notranslate"><span class="pre">val</span></code> specifies its new value.</p>
<p>For example:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="c1"># radosgw-admin zonegroup placement modify --rgw-zonegroup default \</span>
                                           <span class="o">--</span><span class="n">placement</span><span class="o">-</span><span class="nb">id</span> <span class="n">default</span><span class="o">-</span><span class="n">placement</span> \
                                           <span class="o">--</span><span class="n">storage</span><span class="o">-</span><span class="k">class</span> <span class="nc">CLOUDTIER</span> \
                                           <span class="o">--</span><span class="n">tier</span><span class="o">-</span><span class="n">config</span><span class="o">=</span><span class="n">endpoint</span><span class="o">=</span><span class="n">http</span><span class="p">:</span><span class="o">//</span><span class="n">XX</span><span class="o">.</span><span class="n">XX</span><span class="o">.</span><span class="n">XX</span><span class="o">.</span><span class="n">XX</span><span class="p">:</span><span class="n">YY</span><span class="p">,</span>\
                                           <span class="n">access_key</span><span class="o">=&lt;</span><span class="n">access_key</span><span class="o">&gt;</span><span class="p">,</span><span class="n">secret</span><span class="o">=&lt;</span><span class="n">secret</span><span class="o">&gt;</span><span class="p">,</span> \
                                           <span class="n">multipart_sync_threshold</span><span class="o">=</span><span class="mi">44432</span><span class="p">,</span> \
                                           <span class="n">multipart_min_part_size</span><span class="o">=</span><span class="mi">44432</span><span class="p">,</span> \
                                           <span class="n">retain_head_object</span><span class="o">=</span><span class="n">true</span>
</pre></div>
</div>
<p>Nested values can be accessed using period. For example:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span># radosgw-admin zonegroup placement modify --rgw-zonegroup={zone-group-name} \
                                           --placement-id={placement-id} \
                                           --storage-class={storage-class-name} \
                                           --tier-config=acls.source_id=${source-id}, \
                                           acls.dest_id=${dest-id}
</pre></div>
</div>
<p>Configuration array entries can be accessed by specifying the specific entry to be referenced enclosed
in square brackets, and adding new array entry can be done by using <cite>[]</cite>.
For example, creating a new acl array entry:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span># radosgw-admin zonegroup placement modify --rgw-zonegroup={zone-group-name} \
                                           --placement-id={placement-id} \
                                           --storage-class={storage-class-name} \
                                           --tier-config=acls[].source_id=${source-id}, \
                                           acls[${source-id}].dest_id=${dest-id}, \
                                           acls[${source-id}].type=email
</pre></div>
</div>
<p>An entry can be removed by using <code class="docutils literal notranslate"><span class="pre">--tier-config-rm={key}</span></code>.</p>
<p>For example,</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="c1"># radosgw-admin zonegroup placement modify --rgw-zonegroup default \</span>
                                           <span class="o">--</span><span class="n">placement</span><span class="o">-</span><span class="nb">id</span> <span class="n">default</span><span class="o">-</span><span class="n">placement</span> \
                                           <span class="o">--</span><span class="n">storage</span><span class="o">-</span><span class="k">class</span> <span class="nc">CLOUDTIER</span> \
                                           <span class="o">--</span><span class="n">tier</span><span class="o">-</span><span class="n">config</span><span class="o">-</span><span class="n">rm</span><span class="o">=</span><span class="n">acls</span><span class="o">.</span><span class="n">source_id</span><span class="o">=</span><span class="n">testid</span>

<span class="c1"># radosgw-admin zonegroup placement modify --rgw-zonegroup default \</span>
                                           <span class="o">--</span><span class="n">placement</span><span class="o">-</span><span class="nb">id</span> <span class="n">default</span><span class="o">-</span><span class="n">placement</span> \
                                           <span class="o">--</span><span class="n">storage</span><span class="o">-</span><span class="k">class</span> <span class="nc">CLOUDTIER</span> \
                                           <span class="o">--</span><span class="n">tier</span><span class="o">-</span><span class="n">config</span><span class="o">-</span><span class="n">rm</span><span class="o">=</span><span class="n">target_path</span>
</pre></div>
</div>
<p>The storage class can be removed using the following command</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="c1"># radosgw-admin zonegroup placement rm --rgw-zonegroup={zone-group-name} \</span>
                                       <span class="o">--</span><span class="n">placement</span><span class="o">-</span><span class="nb">id</span><span class="o">=</span><span class="p">{</span><span class="n">placement</span><span class="o">-</span><span class="nb">id</span><span class="p">}</span> \
                                       <span class="o">--</span><span class="n">storage</span><span class="o">-</span><span class="n">class</span><span class="o">=</span><span class="p">{</span><span class="n">storage</span><span class="o">-</span><span class="n">class</span><span class="o">-</span><span class="n">name</span><span class="p">}</span>
</pre></div>
</div>
<p>For example,</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="c1"># radosgw-admin zonegroup placement rm --rgw-zonegroup default \</span>
                                       <span class="o">--</span><span class="n">placement</span><span class="o">-</span><span class="nb">id</span> <span class="n">default</span><span class="o">-</span><span class="n">placement</span> \
                                       <span class="o">--</span><span class="n">storage</span><span class="o">-</span><span class="k">class</span> <span class="nc">CLOUDTIER</span>
<span class="p">[</span>
    <span class="p">{</span>
        <span class="s2">&quot;key&quot;</span><span class="p">:</span> <span class="s2">&quot;default-placement&quot;</span><span class="p">,</span>
        <span class="s2">&quot;val&quot;</span><span class="p">:</span> <span class="p">{</span>
            <span class="s2">&quot;name&quot;</span><span class="p">:</span> <span class="s2">&quot;default-placement&quot;</span><span class="p">,</span>
            <span class="s2">&quot;tags&quot;</span><span class="p">:</span> <span class="p">[],</span>
            <span class="s2">&quot;storage_classes&quot;</span><span class="p">:</span> <span class="p">[</span>
                <span class="s2">&quot;STANDARD&quot;</span>
            <span class="p">]</span>
        <span class="p">}</span>
    <span class="p">}</span>
<span class="p">]</span>
</pre></div>
</div>
</div>
</div>
<div class="section" id="object-modification-limitations">
<h2>Object modification &amp; Limitations<a class="headerlink" href="#object-modification-limitations" title="Permalink to this headline">¶</a></h2>
<p>The cloud storage class once configured can then be used like any other storage class in the bucket lifecyle rules. For example,</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="o">&lt;</span><span class="n">Transition</span><span class="o">&gt;</span>
  <span class="o">&lt;</span><span class="n">StorageClass</span><span class="o">&gt;</span><span class="n">CLOUDTIER</span><span class="o">&lt;/</span><span class="n">StorageClass</span><span class="o">&gt;</span>
  <span class="o">....</span>
  <span class="o">....</span>
<span class="o">&lt;/</span><span class="n">Transition</span><span class="o">&gt;</span>
</pre></div>
</div>
<p>Since the transition is unidirectional, while configuring S3 lifecycle rules, the cloud storage class should be specified last among all the storage classes the object transitions to. Subsequent rules (if any) do not apply post transition to the cloud.</p>
<p>Due to API limitations there is no way to preserve original object modification time and ETag but they get stored as metadata attributes on the destination objects, as shown below:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">x</span><span class="o">-</span><span class="n">amz</span><span class="o">-</span><span class="n">meta</span><span class="o">-</span><span class="n">rgwx</span><span class="o">-</span><span class="n">source</span><span class="p">:</span> <span class="n">rgw</span>
<span class="n">x</span><span class="o">-</span><span class="n">amz</span><span class="o">-</span><span class="n">meta</span><span class="o">-</span><span class="n">rgwx</span><span class="o">-</span><span class="n">source</span><span class="o">-</span><span class="n">etag</span><span class="p">:</span> <span class="n">ed076287532e86365e841e92bfc50d8c</span>
<span class="n">x</span><span class="o">-</span><span class="n">amz</span><span class="o">-</span><span class="n">meta</span><span class="o">-</span><span class="n">rgwx</span><span class="o">-</span><span class="n">source</span><span class="o">-</span><span class="n">key</span><span class="p">:</span> <span class="n">lc</span><span class="o">.</span><span class="n">txt</span>
<span class="n">x</span><span class="o">-</span><span class="n">amz</span><span class="o">-</span><span class="n">meta</span><span class="o">-</span><span class="n">rgwx</span><span class="o">-</span><span class="n">source</span><span class="o">-</span><span class="n">mtime</span><span class="p">:</span> <span class="mf">1608546349.757100363</span>
<span class="n">x</span><span class="o">-</span><span class="n">amz</span><span class="o">-</span><span class="n">meta</span><span class="o">-</span><span class="n">rgwx</span><span class="o">-</span><span class="n">versioned</span><span class="o">-</span><span class="n">epoch</span><span class="p">:</span> <span class="mi">0</span>
</pre></div>
</div>
<p>By default, post transition, the source object gets deleted. But it is possible to retain its metadata but with updated values (like storage-class and object-size) by setting config option ‘retain_head_object’ to true. However GET on those objects shall still fail with ‘InvalidObjectState’ error.</p>
<p>For example,</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="c1"># s3cmd info s3://bucket/lc.txt</span>
<span class="n">s3</span><span class="p">:</span><span class="o">//</span><span class="n">bucket</span><span class="o">/</span><span class="n">lc</span><span class="o">.</span><span class="n">txt</span> <span class="p">(</span><span class="nb">object</span><span class="p">):</span>
   <span class="n">File</span> <span class="n">size</span><span class="p">:</span> <span class="mi">12</span>
   <span class="n">Last</span> <span class="n">mod</span><span class="p">:</span>  <span class="n">Mon</span><span class="p">,</span> <span class="mi">21</span> <span class="n">Dec</span> <span class="mi">2020</span> <span class="mi">10</span><span class="p">:</span><span class="mi">25</span><span class="p">:</span><span class="mi">56</span> <span class="n">GMT</span>
   <span class="n">MIME</span> <span class="nb">type</span><span class="p">:</span> <span class="n">text</span><span class="o">/</span><span class="n">plain</span>
   <span class="n">Storage</span><span class="p">:</span>   <span class="n">CLOUDTIER</span>
   <span class="n">MD5</span> <span class="nb">sum</span><span class="p">:</span>   <span class="n">ed076287532e86365e841e92bfc50d8c</span>
   <span class="n">SSE</span><span class="p">:</span>       <span class="n">none</span>
   <span class="n">Policy</span><span class="p">:</span>    <span class="n">none</span>
   <span class="n">CORS</span><span class="p">:</span>      <span class="n">none</span>
   <span class="n">ACL</span><span class="p">:</span>       <span class="n">M</span><span class="o">.</span> <span class="n">Tester</span><span class="p">:</span> <span class="n">FULL_CONTROL</span>
   <span class="n">x</span><span class="o">-</span><span class="n">amz</span><span class="o">-</span><span class="n">meta</span><span class="o">-</span><span class="n">s3cmd</span><span class="o">-</span><span class="n">attrs</span><span class="p">:</span> <span class="n">atime</span><span class="p">:</span><span class="mi">1608466266</span><span class="o">/</span><span class="n">ctime</span><span class="p">:</span><span class="mi">1597606156</span><span class="o">/</span><span class="n">gid</span><span class="p">:</span><span class="mi">0</span><span class="o">/</span><span class="n">gname</span><span class="p">:</span><span class="n">root</span><span class="o">/</span><span class="n">md5</span><span class="p">:</span><span class="n">ed076287532e86365e841e92bfc50d8c</span><span class="o">/</span><span class="n">mode</span><span class="p">:</span><span class="mi">33188</span><span class="o">/</span><span class="n">mtime</span><span class="p">:</span><span class="mi">1597605793</span><span class="o">/</span><span class="n">uid</span><span class="p">:</span><span class="mi">0</span><span class="o">/</span><span class="n">uname</span><span class="p">:</span><span class="n">root</span>

<span class="c1"># s3cmd get s3://bucket/lc.txt lc_restore.txt</span>
<span class="n">download</span><span class="p">:</span> <span class="s1">&#39;s3://bucket/lc.txt&#39;</span> <span class="o">-&gt;</span> <span class="s1">&#39;lc_restore.txt&#39;</span>  <span class="p">[</span><span class="mi">1</span> <span class="n">of</span> <span class="mi">1</span><span class="p">]</span>
<span class="n">ERROR</span><span class="p">:</span> <span class="n">S3</span> <span class="n">error</span><span class="p">:</span> <span class="mi">403</span> <span class="p">(</span><span class="n">InvalidObjectState</span><span class="p">)</span>
</pre></div>
</div>
<p>To avoid object names collision across various buckets, source bucket name is prepended to the target object name. If the object is versioned, object versionid is appended to the end.</p>
<p>Below is the sample object name format:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">s3</span><span class="p">:</span><span class="o">//&lt;</span><span class="n">target_path</span><span class="o">&gt;/&lt;</span><span class="n">source_bucket_name</span><span class="o">&gt;/&lt;</span><span class="n">source_object_name</span><span class="o">&gt;</span><span class="p">(</span><span class="o">-&lt;</span><span class="n">source_object_version_id</span><span class="o">&gt;</span><span class="p">)</span>
</pre></div>
</div>
<div class="section" id="versioned-objects">
<h3>Versioned Objects<a class="headerlink" href="#versioned-objects" title="Permalink to this headline">¶</a></h3>
<p>For versioned and locked objects, similar semantics as that of LifecycleExpiration are applied as stated below.</p>
<ul class="simple">
<li><p>If the object is current, post transitioning to cloud, it is made noncurrent with delete marker created.</p></li>
<li><p>If the object is noncurrent and is locked, its transition is skipped.</p></li>
</ul>
</div>
</div>
<div class="section" id="future-work">
<h2>Future Work<a class="headerlink" href="#future-work" title="Permalink to this headline">¶</a></h2>
<ul class="simple">
<li><p>Send presigned redirect or read-through the objects transitioned to cloud</p></li>
<li><p>Support s3:RestoreObject operation on cloud transitioned objects.</p></li>
<li><p>Federation between RGW and Cloud services.</p></li>
<li><p>Support transition to other cloud provideres (like Azure).</p></li>
</ul>
</div>
</div>



           </div>
           
          </div>
          <footer>
    <div class="rst-footer-buttons" role="navigation" aria-label="footer navigation">
        <a href="../../mgr/" class="btn btn-neutral float-right" title="Ceph 管理器守护进程" accesskey="n" rel="next">Next <span class="fa fa-arrow-circle-right" aria-hidden="true"></span></a>
        <a href="../d3n_datacache/" class="btn btn-neutral float-left" title="D3N RGW Data Cache" accesskey="p" rel="prev"><span class="fa fa-arrow-circle-left" aria-hidden="true"></span> Previous</a>
    </div>

  <hr/>

  <div role="contentinfo">
    <p>
        &#169; Copyright 2016, Ceph authors and contributors. Licensed under Creative Commons Attribution Share Alike 3.0 (CC-BY-SA-3.0).

    </p>
  </div> 

</footer>
        </div>
      </div>

    </section>

  </div>
  

  <script type="text/javascript">
      jQuery(function () {
          SphinxRtdTheme.Navigation.enable(true);
      });
  </script>

  
  
    
   

</body>
</html>